Investigating the Practical Consequences of Model Misfit in Unidimensional IRT Models

In this article, the practical consequences of violations of unidimensionality on selection decisions in the framework of unidimensional item response theory (IRT) models are investigated based on simulated data. The factors manipulated include the severity of violations, the proportion of misfitting items, and test length. The outcomes that were considered are the precision and accuracy of the estimated model parameters, the correlations of estimated ability ([Formula: see text]) and number-correct ([Formula: see text]) scores with the true ability ([Formula: see text]), the ranks of the examinees and the overlap between sets of examinees selected based on either [Formula: see text], [Formula: see text], or [Formula: see text] scores, and the bias in criterion-related validity estimates. Results show that the [Formula: see text] values were unbiased by violations of unidimensionality, but their precision decreased as multidimensionality and the proportion of misfitting items increased; the estimated item parameters were robust to violations of unidimensionality. The correlations between [Formula: see text], [Formula: see text], and [Formula: see text] scores, the agreement between the three selection criteria, and the accuracy of criterion-related validity estimates are all negatively affected, to some extent, by increasing levels of multidimensionality and the proportion of misfitting items. However, removing the misfitting items only improved the results in the case of severe multidimensionality and large proportion of misfitting items, and deteriorated them otherwise.